Fix Occasional Long Operations #84
Merged
+646
−706
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Through some debugging it looks like we were not handling connection pool logic properly causing a large amount of stale connections to exist on the database.
My hypothesis is that occasionally the database would saturate causing it to take up to 15 minutes for a connection to be available to complete even simple database operations like
get_or_create_app
I used some of the below queries on Postgres to check for connections.
Check for idle connections older than an hour
Check for all idle connections
Stats on database connection capacity
Originally before the changes I would see upwards of 45 connections, now it is closer to 14. I also do not see any idle connections anymore.